Description

Load one shared stream service per in-process model worker while retaining multiple independent LiveKit room and pipeline sessions.

This change adds retained-session admission capacity and a LingBot execution lease that serializes model work across sessions at chunk boundaries. It also aligns worker health aggregation, controller activity handling, CLI configuration, examples, and documentation with the multi-session lifecycle.

The default retained-session capacity remains 1, preserving existing behavior unless multi-session operation is explicitly enabled.

Motivation

Previously, one LiveKit worker effectively handled one room session at a time. Supporting multiple users by loading a separate model instance per session would duplicate model weights and significantly increase GPU memory consumption.

This change allows multiple user sessions to share one loaded model service while preserving independent room, control, cache, noise, VAE, and pipeline-session state. When multiple sessions request execution, the LingBot service grants one execution lease and hands it off at a chunk boundary after the active controller becomes idle.

This provides bounded time-division multiplexing without introducing continuous batching, cross-session state sharing, or additional model replicas.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Performance improvement
• Code refactoring
• Documentation update
• Other (please describe):

Changes Made

• Added per-worker retained-session capacity with bounded HTTP admission queuing.
• Added a multi-session LiveKit worker that loads one shared stream service and runs independent room/session runners.
• Added a LingBot execution-lease manager with control-idle detection, FIFO waiting, chunk-boundary handoff, and cleanup.
• Preserved independent pipeline, cache, control, noise, and VAE state for every admitted LingBot session.
• Added periodic controller heartbeats while browser controls remain held.
• Fixed aggregate worker status, health reporting, shutdown handling, and retained-session release semantics.
• Added CLI options and configuration for max_sessions_per_worker and control_idle_timeout.
• Clarified service process, model worker, service instance, session runner, pipeline session, queue, GPU placement, and health semantics in English and Chinese documentation.
• Made localhost server tests independent of process-level HTTP proxy settings.
• Added focused execution-lease, capacity, multi-session worker, runtime, CLI, demo, and action-loop tests.

Testing

• Unit tests pass using the repository CI marker selection
• Manual testing performed with two independent LiveKit sessions on four H100 GPUs
• Benchmarks added/updated (if applicable)

Test commands:

ruff check telefuser tests --output-format=full
ruff format --check telefuser tests
ruff check --select I telefuser tests

pytest tests/unit -v \
  -m "not gpu and not distributed and not slow and not quant" \
  --tb=short

pytest tests/server/ -v \
  -m "not gpu and not distributed and not slow" \
  --tb=short

mkdocs build --strict --site-dir /tmp/telefuser-ci-docs
git diff --check

Results:

• Ruff and import checks passed.
• 857 CI-selected unit tests passed; 308 were deselected by CI markers.
• 62 server tests passed.
• English and Chinese MkDocs sites built successfully in strict mode.
• Manual two-session LiveKit testing completed successfully.
• Pre-commit Ruff and formatting hooks passed during commit.

Checklist

• Code follows the project's coding standards (ruff)
• Commit-time pre-commit hooks pass
• CI-selected unit and server tests pass
• New tests added for new functionality
• Documentation updated
• Commit messages are clear and descriptive
• PR title follows the convention: [TYPE] Brief description

Related Issues

None.

Additional Notes

• max_sessions_per_worker defaults to 1, so existing deployments retain their previous behavior.
• Multi-session capacity requires a BidirectionalService; server-push capacity greater than one is rejected.
• The LiveKit runtime currently supports exactly one in-process model worker. Additional model replicas still require separate stream-serve processes and external routing.
• control_idle_timeout only controls execution-lease handoff when another session is waiting. It does not close the idle session, release its admission slot, or free its per-session cache.
• Parked sessions retain their model state and GPU memory. Capacity should therefore be configured from measured memory headroom.
• This is chunk-boundary time slicing, not continuous batching.

GPU Architecture Support

Not applicable. This PR does not add or modify CUDA or Triton kernels.

• SM80 (Ampere, Ada Lovelace)
• SM90 (Hopper H100)
• SM100+ (Blackwell)

Performance Impact

The change avoids loading one model service instance per user session. Multiple retained sessions share model weights while keeping session-specific runtime state isolated.

Only one LingBot session chunk executes at a time, so the change does not increase model compute concurrency or claim a throughput improvement. Lease handoff introduces only control-path synchronization at chunk boundaries.

Parked sessions retain per-session cache and pipeline state, so GPU memory usage grows with retained-session count. Two-session operation was manually validated on four H100 GPUs; no new formal benchmark is included.